Drawings, sketches, artwork, photographs, and other images have long been recognized to have significant commercial and/or personal value. Paradoxically, as such images age, they become both more valuable and less valuable. Their increased value reflects their success in capturing events, emotions, people, objects, achievements, etc. that have attain enhanced personal, historical or commercial significance over the passage of time. The paradoxical diminution of their value reflects the inexorable loss of information regarding the people, events, or scenes shown in the images that occurs when those having actual knowledge of the depicted images become unavailable or die. This phenomenon, as applied to personal images (e.g., photographs, etc.), has been referred to as the “Personal Photo Paradox” (or “P3”).
One means for addressing the paradoxical diminution in the value of aging images is to organize the images into scrapbooks or albums, in which the individual images are captioned. Adding captions to photographs is, however, a time-consuming and error prone task even for professional photographers, editors, librarians, curators, and scholars. In many professional applications, photographs are worthless unless they are accurately described by date, time, location, photographer, title, recognizable people, etc. Additional descriptions may include details about the mechanics of the photograph (for example, film type, print size, aperture, shutter speed, owner, copyright information) and/or its contents (keywords from controlled vocabularies, topics from a hierarchy, free text descriptions, etc.). In light of the time required to add such captions, amateur photographers rarely systematically arrange or notate their collections, except for an occasional handwritten note on the back of a photograph or an envelope containing a collection of photographs.
For those who are serious about adding annotations, the common computer-based approach is to use database programs, such as Microsoft® Access®, that offer “form fill-in” or “free” text boxes and then store the information in a database. Data entry is typically done by typing, but many systems permit certain attribute values for some fields (for example, black & white or color film) to be selected by the individual keeping the collection. Of course, simpler tools that provide free-form input, such as word processors, spreadsheets, and other tools are used in many situations. Captions and annotations are often displayed near a photograph on screen displays, web pages, and printed versions (see, e.g., U.S. Pat. Nos. 6,111,586 (Issued Aug. 29, 2000); 5,873,080 (Issued Feb. 16, 1999); 5,493,677 (Issued Feb. 20, 1996); 5,404,435 (Issued Apr. 4, 1995); 5,142,662 (Issued Aug. 25, 1992) and by PCT Application WO00046695A1 (Published Aug. 10, 2000) and European Patent Applications EP00938227A3 (Published Sep. 6, 2000) and EP01006464A2 (Published Jun. 7, 2000). Software packages (Kodak PhotoEasy®, MGI PhotoSuite®, Aladdin Image AXS®, etc.) and web sites (Kodak's photonet, Gatherround.com, shutterfly, etc.) offer modest facilities to typing in annotations and searching descriptions.
As photograph library sizes increase, the need and benefit of annotation and search capabilities grows. The need to rapidly locate photographs of, for example, Bill Clinton meeting with Boris Yeltsin at a European summit held in 1998, is strong enough to justify substantial efforts in many news agencies. More difficult searches such as images depicting “agriculture in developing nations” are harder to satisfy, but many web and database search tools support such searches (Lycos, Corbis, etc.). Query-By-Image-Content from IBM, is one of many projects that use automated techniques to analyze image. Computer vision techniques can be helpful in finding photographs by color (sunsets are a typical example), identifying features (corporate logos or the Washington Monument), or textures (such as clouds or trees), but a blend of automated and manual techniques may be preferable. Approaches to the non-textual or “content-based” annotation of images are described in U.S. Pat. Nos. 5,911,139 (Issued Jun. 8, 1999), 5,899,999 (Issued May 4, 1999), and 5,579,471 (Issued Nov. 26, 1996) and PCT Applications WO09935596A1 (Published Jul. 15, 1999), WO09931605A1 (Published Jun. 24, 1999). The non-textual searching of image files is discussed by Rui, et al. (1999; “Image Retrieval: “Current Techniques Promising Directions and Open Issues”), and by Chang, S.-F. et al. (1997), “Visual Information Retrieval from Large Distributed On-line Repositories”. Face recognition research offers hope for automated annotation, but commercial progress is slow (R. Chellappa, C.L. Wilson and S. Sirohey, “Human and Machine Recognition of Faces: A Survey” Proceedings of the IEEE, Vol. 83, pp. 705–740, May 1995; Allan Kuchinsky, Celine Pering, Michael L. Creech, Dennis Freeze, Bill Serra, Jacek Gwizdka, “FotoFile: A Consumer Multimedia Organization and Retrieval System”, Proceedings of ACM CHI99 Conference on Human Factors in Computing Systems, 496–503, 1999).
The annotation of photographs is a variation on previously explored problems such as annotation on maps (E. Imhof, “Positioning Names on Maps”, The American Cartographer, 2, 128–144, 1975; J. Christensen, J. Marks, and S. Shieber, “An Empirical Study Of Algorithms For Point-Feature Label Placement”, ACM Transactions on Graphics 14, 3, 203–232, 1995; J. S. Doerschler and H. Freeman, “A Rule-Based System For Dense-Map Name Placement”, Communications Of The ACM 35, 1, 68–79, 1992) in which the challenge is to place city, state, river, or lake labels close to the features. There is a long history of work on this problem, but new possibilities emerge because of the dynamics of the computer screen. Such efforts, however, are fundamentally different from those involved in the annotation of photographs, video, and other electronic images in which the image aspects (i.e., the objects, people scenes, depictions, etc. that are contained within an electronic image) are inherently unrelated to one another. Maps (and electronic timelines, are X,Y (or, in the case of timelines, X) representations of a surface or construct in which the relative placements of all desired annotations are determined solely by the scale of the image and the X,Y (or X) coordinate of its center, rather than by inspection and analysis of the image. The annotation of maps and timelines is possible using prior art methods because their image aspects are inherently related to one another. Annotation is usually seen as an authoring process conducted by specialists and users only chose whether to show or hide annotations. Variations on annotation also come from the placement of labels on markers in information visualization tasks such as in tree structures, such in the hyperbolic tree (John Lamping, Ramana Rao, and Peter Pirolli, “A Focus + Context Technique Based On Hyperbolic Geometry For Visualizing Large Hierarchies”, Proceedings of ACM CHI95 Conference on Human Factors in Computing Systems, New York, 401–408, 1995) or in medical histories, such as LifeLines (Jia Li., Catherine Plaisant, Ben Shneidernian, “Data Object and Label Placement for Information Abundant Visualizations” Workshop on New Paradigms in Information Visualization and Manipulation (NPIV'98), ACM, New York, 41–48, 1998).
Previous work on electronic image annotation focused on the writing of computer programs to make label placements that reduced overlaps (Mark D. Pritt, “Method and Apparatus for The Placement of Annotations on A Display without Overlap”, U.S. Pat. No. 5689717, 1997), but there are many situations in which it is helpful for users to place labels manually, much like a post-it® note, on documents, photographs, maps, diagrams, webpages, etc. Annotation of paper and electronic documents by hand is also a much-studied topic with continuing innovations (Bill N. Schilit, Gene Golovchinsky, and Morgan N. Price, “Beyond Paper: Supporting Active Reading with Free Form Digital Ink Annotations,” Proceedings of ACM CHI 98 Conference on Human Factors in Computing Systems, v.1 249–256, 1998). While many systems allow notes to be placed on a document or object, the demands of annotating personal photograph libraries are worthy of special study (J Kelly Lee and Dana Whitney Wolcott, “Method of Customer Photoprint Annotation”, U.S. Pat. No. 5757466, 1998). Personal photograph libraries are considered to represent a special case because users are concentrating on the photographs (and may have a low interested in the underlying technology), are concerned about the social aspects of sharing photographs, and are intermittent users. They seek enjoyment and have little patience for form filling or data entry. Personal photograph libraries may have from hundreds to tens of thousands of photographs, and organization is, to be generous, haphazard. Photographs are sometimes in neat albums, but more often put in a drawer or a shoebox. While recent photographs are often on top, shuffling through the photographs often leaves them disorganized. Some users will keep photographs in the envelopes they got from the photography store, and more organized types will label and order them. The annotating of such personal images is such a time-consuming, tedious and error-prone data entry task that it discourages many archivists and librarians from maximizing the value of their image collections. As noted by Rui, et al. (1999), the perception subjectivity and annotation impreciseness of text-based image retrieval systems may cause unrecoverable mismatches in the later retrieval process. The problem is particularly significant for most owners of personal photograph libraries.
As digital cameras become widespread, users have had to improvise organization strategies using hierarchical directory structures, and typing in descriptive file and directory names to replace the automatically generated photograph file numbers. Some software packages (PhotoSuite, PhotoEasy, etc.) enable users to organize photographs into albums and create web pages with photographs, but annotation is often impossible or made difficult. Web sites such as Kodak's PhotoNet.com, AOL.com, Gatherround.com, etc. enable users to store collections of photographs and have discussion groups about the collections, but annotation is limited to typing into a caption field. The FotoFile software (Allan Kuchinsky, Celine Pering, Michael L. Creech, Dennis Freeze, Bill Serra, Jacek Gwizdka, “FotoFile: A Consumer Multimedia Organization and Retrieval System”, Proceedings of ACM CHI99 Conference on Human Factors in Computing Systems, 496–503, 1999) is considered particularly relevant.
It is an objective of the present invention to provide a computer software program that permits both skilled and intermittent users the ability to annotate electronic images in a manner that is easy and accurate, and which permits the user or others to retrieve, organize, additionally annotate display and/or distribute such images as desired.